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Abstract. In this paper wc study irreducible unitary representations of GLn (K) and prove a number 
of results. Our first result establishes a precise connection between the annihilator of a representation 
and the existence of degenerate Whittaker functionals, for both smooth and K-finite vectors, thereby 
generalizing results of Kostant, Matumoto and others. 

Our second result relates the annihilator to the sequence of adduced representations, as defined in 
this setting by one of the authors. Based on those results, we suggest a new notion of rank of a smooth 
admissible representation of GLn(IR), which for unitarizable representations refines Howe's notion of 
rank. 

Our third result computes the adduced representations for (almost) all irreducible unitary represen- 
tations in terms of the Vogan classification. 

We also indicate briefly the analogous results over complex and p-adic fields. 
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1. Introduction 

Many important problems in harmonic analysis require one to decompose a unitary representation 
of a real reductive group G on a Hilbert space T-L, e.g. H = {X) for some G-space X equipped 
with an invariant measure. In order to solve such problems one would like to know the irreducible 
unitary representations of G as explicitly as possible. The starting point is of course the determination 
of the unitary dual G, but then it is helpful to have additional knowledge about invariants of unitary 
representations, such as their annihilator varieties, existence of Whittaker functionals, etc. 

The unitary dual of G„ = GL (n, M.) has been determined by Vogan |Vog86[ and in this paper we 
consider the following invariants of tt G G„, whose precise definitions are given below in section [1.21 

(1) the annihilator variety V (tt) C 0[(n,C)*, 

(2) the space Wh^^ (tt) of (degenerate) Whittaker functionals of type a, 

(3) the depth composition DC (tt) for iterated adduced representations of tt, 

(4) the Howe rank HR (tt) of tt. 

Our main theorem generalizes a number of existing results in the literature. 

Theorem A. Let n G G„ and let A be the partition of n such that V(7r) = 0\. Then 
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(1) Whl (^) ^ 0. 

(2) DC{tt) = A. 



In particular, this implies that the depth composition is non-increasing. By Matumoto's theorem (see 
Corollarv ll.4.1l below) our result implies also that A is the biggest partition with Wh*^ (tt) ^ 0. By |He08[ 
Theorems 0.1, 0.2, 4.3], A connects to Howe's notion of rank by 

HR (tt) = min ( [n/2j , n - length{X)) . 

We will give an independent proof of this result (see Remark I4.2.3P . 

Before giving the precise definitions of our invariants, we need to fix some notation regarding partitions, 
compositions, nilpotent orbits, and parabolic subgroups. 

1.1. Notation. 

Definition 1.1.1. A composition of n of length k is a sequence a = {ai, ...^ ak) of natural numbers 
(i.e. strictly positive integers) such that Scti = n; a partition is a nondecreasing composition. For a 
composition a we denote by a-/a~ the nondecreasing /nonincreasing reordering of a. 

Remark 1.1.2. The Young diagram of a partition is a left alligned array of boxes with A^ boxes in row 
i. The rows of the diagram for A* are the columns of the diagram for A. 

We sometimes use "exponential" notation for partitions; thus 4^2^1'^ denotes (4,4, 2, 1, 1, 1). 
We denote 0„ := flI„(C). 

Definition 1.1.3. If a is a composition of n, we define Ja G Qn to be the Jordan matrix with diagonal 
Jordan blocks of size ai,...,afc; explicitly 



We define Oa to be the orbit of Ja under the adjoint action of GL (n, C) on Qn. 

If A = a- then we have Oa = 0\. Moreover by the theorem of Jordan canonical form, for each 
nilpotent matrix X in g„ there is a unique partition A such that X G 0\. 

Let A, /i be partitions of n and let 0\ denote the Zariski closure of 0\ then we have 

Op ^ ^ if Ml + • • • Mfc < Ai H + Afe for all fc; 

If A,/^ satisfy this condition, will simply write C A. 

Remark 1.1.4. For X.,Y in g„ the trace pairing is defined to be 



Then (•, •) is a nondegenerate symmetric GL (n, C)-invariant bilinear form (trace form) on Qn- The trace 
form gives rise to an isomorphism Qn ~ Qn that intertwines the adjoint and coadjoint actions of GL (n, C). 
This allows us to identify adjoint orbits and coadjoint orbits. 

We next fix our conventions regarding parabolic subgroups of G„. 
If a = (ai, ttfe) is a composition of n then we define 

Si = Si [a) := {ai H h + j : I < j < 

For g G Gn we let denote the ai x aj submatrix of g with rows from Si and columns from Sj . 

Definition 1.1.5. For a composition a we define subgroups Pa, La, Na of Gn as follows 



Here 5^^ is Kronecker's 5, while and 1 denote zero and identity matrices of appropriate size. 

Thus B = Pin is the standard Borel subgroup of upper triangular matrices, and each Pa is a standard 
parabolic subgroup containing B. Na is the nilradical of Pa and Pa = LaNa is a Levi decomposition 

with La ~ Gai X • • • X Ga^. ■ 

We now introduce the Bernstein- Zelevinsky product notation for parabolic induction. 




(1) 



{X, Y) = trace {XY) 



Pa^{g\ = ^/^ < ]],La = {g I .9L^ = ^/^ ^ j} ,7V„ = {g \ g^^ = 6'^ if i < j} . 
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Definition 1.1.6. If a = (ai, ...,ak) is a composition of n and ni e Gq. then tti eg) • • • ® tt^ is an 
irreducible unitary representation of La ~ Gai x • • • x Gq,^ . We extend this to Pa trivially on 7V„ and 
define 

TTl X • • ■ X TTfc = Indp2 (tTi • • • ® TTfe) , 

where Ind denotes normalized induction (see ^ ^2.2\) . 

Remark 1.1.7. It follows from |Vog86| or from f(Sah89] and [Bar03| J that if -Ki g Ga^ then tti x • • • x tt/j S 
G„. In this case tti x ■ ■ ■ x nk is unchanged under permutation of the tt^. 

Remark 1.1.8. Since G„/Pq is compact, one can define tti x • ■ • x tt^ analogously in the C°° category. 
We refer the reader to |Vog81| for details. We will occasionally need to consider this case especially in 
connection with complementary series construction in the next section and elsewhere. 

1.2. Invariants of unitary representations. For a representation tt of a Lie group G in a Hilbcrt 
space, we denote by tt°° the space of smooth vectors and by tt" the space of analytic vectors (see S tj2.3p . 
If G is a real reductive group with maximal compact subgroup K we also consider the Harish-Chandra 
module n^'^ consisting of if-finite vectors. 

By [Wall88[ Theorem 3.4.12] if tt is an irreducible unitary representation of a reductive group G with 
Lie algebra g and maximal compact subgroup K, then is an irreducible (g, A')-module and thus tt°° 
is a topologically irreducible representation of G. 

1.2.1. The annihilator variety and associated partition. For an associative algebra A the annihilator of a 
module (cr, V) is 

Ann{(T) ^ {a e A: a {a) V ^ ioT all V e V} 
If A is abelian then we define the annihilator variety of cr to be the variety corresponding to the ideal 
Ann{a), i.e. V (ct) = Zeroes (Ann (ct)). 

If (cr, V) is a module for a Lie algebra g, then one can apply the above considerations to the enveloping 
algebra U (g). While U (g) is not abelian it admits a natural filtration such that gr (U (g)) is the symmetric 
algebra S (g) , and one has a "symbol" map gr from U (g) to S (g). We let gr {Ann{a)) be the ideal in 
S (g) generated by the symbols {gr (a) | a G Ann (cr)} and define the annihilator variety of cr to be 

V (cr) = Zeroes [gr {Ann{a))) C g* 

If g is a complex reductive Lie algebra and M is an irreducible g-module, then it was shown by Joseph 
(see |Jos85j ) that V(M) is the closure O of a single nilpotent coadjoint orbit O. 

If TT is a Hilbert space representation of a Lie group G then we define V(7r) :— V(7r"). If G is reductive 
and TT is an admissible (e.g. irreducible unitary) representation then tt^'-^ C tt" C 7r°°. tt^*-^ is dense in 
■K°° and the action of [/(g) is continuous. Thus, V{-k) = V{tt^'~^). 

If TT is an irreducible unitary representation then tt^'-^ is an irreducible (g, i4r)-module and thus is a 
finite direct sum of algebraically irreducible representations of g. These representations are if-conjugate 
and thus have the same annihilator variety. Thus |Jos85j implies that V{tt) consists of a single nilpotent 
coadjoint orbit, that we call the associated orbit. 

Definition 1.2.2. If X is a partition of n such that V(7r) = 0\ we call A associated partition and denote 
A = AP{ti). 

For example, if tt is finite-dimensional then V(7r) — {0} and AP(7r) = 1" and if tt is generic then, by 
a result of Kostant (see iji jl.4[) . V(7r) is the nilpotent cone of g* and AP(7r) ~ n^. 

For an admissible representation tt of a real reductive group G with Lie algebra gR and complexified 
Lie algebra g one can define more refined invariants such as 

(1) the asymptotic support AS (tt) C g^ 

(2) the wave front set WF (tt) C g^ (see e.g. |SV00j 

(3) the associated variety AV (tt) c £^ C g* (see e.g. |BV78| ). Here t denotes the complexified Lie 
algebra of the maximal compact subgroup. 

By |Ros95j and |SVOO| . these three invariants determine each other and each of them determines V (tt). 
For GL (n, M) the converse is true as well, and since we are primarily interested in this case we will mostly 
ignore the refined invariants in this paper. 
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1.2.3. Degenerate Whittaker Junctionals. In this section we fix n, and write N for A^(in); thus N is the 
subgroup of Gn consisting of unipotent upper triangular matrices. Let riR be the Lie algebra of N and 
let n be the complexification of ur. 

Let 4* denote the set of multiplicative unitary characters of N. Then ^' can be identified with a subset 
of n* via the exponential map. More precisely we have 

* w {V e n* I ([n, n]) = 0, V (hr) C iR} 

where an element -0 of the right side is regarded as character of N via the formula 

ip (expX) = e'^(^) for X e ur. 

We will write for the one-dimensional space regarded as a module for or n via ip. 

Definition 1.2.4. If n E Gn and -0 6 5* u;e define 

Wh'^ (tt) = Homn , Wh*^ (tt) = Hom^°"*(^°°, Q,) 

where Hom^"* denotes the space of continuous N-homomorphisms. 

It is well known that tt"^ is dense in tt°°, hence by restriction we get an inclusion 

(2) Wh*^ (tt) C Wh'^ (tt) 
Moreover tt^'~^ is finitely generated as an n-module and hence we get 

dimWh*^ (tt) < dimWh'^ (tt) < oo 

We refer to elements of Wh*^^ (tt) and Wh'^^ (tt) as (degenerate) Whittaker functionals of type ip. 

Remark. Let n be the space of strictly lower triangular matrices. Then the trace form restricts to a 
nondegenerate pairing ofn with n, allowing one to identify fi ~ n*. Under this identification elements of 
^' correspond to imaginary "subdiagonaV matrices, i.e to matrices X G g„ satisfying 

Xpq £ zM if p ~ q + I and Xpq = if p ^ q + 1 

Since we have identified g* « 0„ via the trace form, the above remark also allows us to regard n* as a 
subspace of g* . Hence we may also regard as a subspace of g* . 

Definition 1.2.5. Let a be a composition ofn, Ja be the corresponding .Jordan matrix as in Definition 
\L1.3\. and w be the longest element of the Weyl group. Then wJaW~^ is subdiagonal and iwJaW~^ can be 
regarded as an element of ^' by the above remark. We denote this character by ipa . Note that tpa G Ca • 
For TT e Gn denote also 

(3) Wh'^ (it) = Wh'^^ (tt) , Whl (tt) ^ Wh;^ (it) 

1.2.6. The adduced representation, "derivatives "and the depth composition. In |BZ77[ lBer84| Bernstein 
and Zelevinsky introduced the important notion of "derivative" for representations of GL {n,Qp). An 
Archimedean analog of the "highest" derivative for tt G G„ was defined in |Sah89| , where it was called the 
adduced representation and denoted An. This definition which we now recall, involves two ingredients. 

Let Pn C Gn be the "mirabolic" subgroup consisting of matrices with last row (0, 0, 0, 1). To 
forestall confusion we note that P„ is not a parabolic subgroup, it has codimension 1 in P(n-i.i) ^-i^d is 
completely different from P(„) ~ Gn. 

The first ingredient in the definition of An is the following result that was conjectured by Kirillov. 

Theorem 1.2.7. Let n G Gn, then n\p^^ is irreducible. 

This was first proven in the p-adic case in |Ber84| . then in the complex case in |Sah89| . and finally in the 
real case in |Bar03| . Three new proofs have been obtained recently in jAG091 ISZl IGaLaj . 

The second ingredient is Mackcy theory that describes the unitary dual of Lie groups, such as P„, 
which are of the form G = H t< Z with Z abelian. In this case Z consists of unitary characters and H 
acts on Z] for x G l^t denote its stabilizer in H. If cr G Sy- then cr ig) x is a unitary representation 
of Z, and we define 

Ix M = Irid^^^z (c^® X) 
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The main results of Mackey theory are as foUows: 

(a) (cr) is irreducible for all a e S^; 

(b) G is the disjoint union of ("^x) ''^^ X ranges over representatives of distinct i7-conjugacy classes in 
Z. 

Since P„ w G„_i k R"~^ we may analyze P„ by Mackey theory. There arc two Gn-i-conjugacy 
classes in M"~i; one class consists of the trivial character xo alone, while the other class contains all other 
characters. As a representative of the second class we pick the character xi defined by 

Xi(ai, ...,a„_i) = exp (ia„_i) 

The stabilizers in G„_i are = G„_i and S^-^ — Pn-i and therefore we get 

(4) K = ho{GZ:i)]\h.(p^i) 

We may iterate (0]) until we arrive at the trivial group Pi = Go 

n 
k=l 

Combining this with Theorem II . 2 . 71 we deduce that for tt G G„ with n > 0, there exists a unique natural 
number d and a unique tt' S Gn-d such that 

(5) ^|p„ = I^-'ho W) 

Definition 1.2.8. jSah89j If tt & G„ and tt' G Gn-d satisfy ^ we say that tt has depth d and that tt' is 
the adduced representation (or highest derivative) of "k , and we write n' — An. 

Let * denote the trivial representation of the trivial group Go- The procedure of taking the adduced 
representation can be iterated until we arrive at Thus we obtain a sequence of unitary representations 

(6) (tto = tt, TTi, . . . , 7ri_i, TT; = *) satisfying ttj = A (tt^-i) for 1 < j < /. 

and we write di = di (tt) for the depth of 7ri_i. Note that d\^di., . . . ^di are natural numbers, and their 
sum is precisely n. 

Definition 1.2.9. The composition (di,d2, ■ ■ ■ ,di) is called the depth composition o/ tt G Gn and is 
denoted DC (tt). 

1.2.10. Howe Rank. Pa is called a maximal parabolic if a has length 2 so that a = (a, b) with a+b — n. In 
this case Na is abelian and isomorphic to Maxb, the additive group of a x 6 real matrices. The unitary dual 
of Na consists of unitary characters and can also be identified with Maxb via Xy i^) = exp (iTr (xy*)). 
The group La = Ga x Gt acts on Maxb in the usual manner, and the orbits are TZq, . . . , 7^min(a,6) where 
TZk denotes the set of matrices of rank k. Note that min (a, 6) < [ri/2j and that equality holds for 
a = [n/2j ,b = n- [n/2\. 

We now briefly describe the theory of Howe rank for G„; let us fix a = (a, n — a) as above. If tt G G„ 
then by Stone's theorem the restriction Trl^r^ corresponds to a projection-valued Borel measure fi^ on 
Na ~ Max{n-a)- Sincc TT is a representation of Pa, fJ-^ is Pa-invariant and decomposes as a direct sum 

= A^O + ■ • ■ + Mmin(a,„-a) with f^l {E) ^ fl^ {E (1 7^fc) 

Building on the work of Howe }How82j . Scaramuzzi [Sca90| proved that /z^ has "pure rank", i.e. there is 
some integer k = HR (tt, a) such that /i'^ = ji^. Moreover if we define HR (tt) = HR (tt, [7T./2J) then 

HR (tt, a) = min [HR (tt) , a, n — a) for all a < n 
Definition 1.2.11. For tt G G„ the integer HR{'k) is called the Howe rank ofn. 
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1.3. Results over other local fields and a uniform formulation of Theorem [Al Theorem \X\ also 
holds in the complex case, and the proof is very similar. We comment on that in section [5] 

In section [6] we prove a p-adic analog of Theorem [21 using |BZ77I IMW87[ IZelSOj . In the p-adic 
case one cannot define annihilator variety, but one can consider the wave front set WFlir). It is not 
known to be the closure of a single nilpotcnt orbit for general reductive group but for irreducible smooth 
representations of GLn{F) this is proven to be the case in |MW87| . 

Since in the p-adic case the notion of derivative is defined for all smooth representations, this analog 
does not require the representation to be unitary. This gives us the following uniform formulation of the 
main theorem. 

Theorem B. Let F he a local field of characteristic zero. Let tt he an irreducihle smooth admissible 
representation of Gh{n^F)) and let WF{'k) C q{^{F) denote the wave front set of tt. Let X be the 
partition of n such that WF(t:) — 0\. Suppose that either F is non- Archimedean or tt is unitarizahle. 
Then 

(1) Wh\{T:) ^ 0, and for any composition a with H^/i* (tt) ^ Q we have Oa C 0\. 

(2) DC (tt) = A; in particular DC (tt) is a non-increasing sequence. 

(3) A is the transpose of the classification partition of t: (see Remark \1.3.1\ below). 
Moreover, if t: is unitarizahle then HR{tt) = min([n/2j ,n ~ length{X)). 

If F is non- Archimedean we also have dimWh*jjf-,^^^{7T) = 1. 

In the p-adic case, Wh'^ denotes the space of all linear equivariant functionals, since all representations 
are considered in discrete topology. 

Remark 1.3.1. The classification partition is defined in [Ven05| for all irreducihle unitary representa- 
tions, through the Tadic-Vogan classification, and in [OS09| for all irreducible smooth representations 
through the Zelevinsky classification. For representations of Arthur type, this partition describes the 
SL{2)-type of the representation of the Weil-Deligne group corresponding to tt by local Langlands corre- 
spondence. 

In the Archimedean case. Theorem |B] follows from Theorem [21 Corollary 11.4.11 Theorem 14.2.11 and 
Remark 14.2.31 below. 

1.4. Earlier works. It was proven by Casselman-Zuckerman for G„ (unpublished) and by Kostant 
f [Kos78| ) for all quasi-split reductive groups that for a generic character -0 of TV, Wh'!^{TT) 7^ if and 
only if Wh'^irr) 7^ if and only if V{tt) is maximal possible, i.e. equal to the nilpotent cone. Matumoto 
f [Mat87l Theorem 1]) proved a generalization of one direction of this statement. For the case of Cn his 
theorem implies 

Corollary 1.4.1. If a is a composition of n, and M is a Qn-module such that IIomn(M, ^/j^) ^ then 
Oo.cV{M). 

Over p-adic fields, a connection between wave front set and generalized Whittaker functionals was 
investigated in |MW87| for smooth (not necessary unitarizahle) representations of any reductive group. 
However, the main theorem of |MW87] involves a lot of choices and for the case of GL„ can be made 
much more concrete, using derivatives and Zelevinsky classification, following jBZ77[ [Zel80j . 

Several works (e.g. |G W80[ iMatOOl IMat88| ) give a partial Archimedean analog of the results of !MW87| . 
by considering non-degenerate ( "admissible" ) characters of the (smaller) nilradicals of bigger parabolic 
subgroups. However, much less is known in the Archimedean case. Our work establishes a different type 
of analog: instead of considering non-degenerate characters of smaller nilradicals, we consider degenerate 
characters of the nilradical of the standard Borel subgroup. Following Zelevinsky jZel801 §§8.3], we call 
functionals equivariant with respect to such characters degenerate Whittaker functionals. 

1.5. Generalizations in future works. In [AGSj . we define the notion of highest derivative for all 
admissible smooth Frechet representations of G„. We show that this notion extends the notion of ad- 
duced representation discussed in the current paper, and establish several properties of highest derivative 
analogous to the ones proven in the p-adic case in jBZ77j . We apply those properties to questions raised 
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in the current paper. Namely, we complete the computation of adduced representations for all Speh 
complementary series, and prove that dim M^/i^pj^j^^ (tt) = 1 for all tt G G„. 
In our work in progress [GSj we prove that 

Whl (tt) ^ ^ Wh'^ (^^^) ^ ^ 0„ C V{tt). 

for any irreducible admissible smooth Frechet representation tt of G„ and any composition a of n. 

Furthermore, we prove the following generalization for any quasi-split real reductive group G. Let N 
be the nilradical of a Borel subgroup of G, and K be maximal compact subgroup of G. Let g and n be 
the complexified Lie algebras of G and A'^. Let tt be an irreducible admissible smooth representation of 
G. Denote 

«'(7r) {V^ G f s.t. Wh*^{n) ^ 0}, and «'(7r^"^) {i' G s.t. Wk'^iir) ^ 0}. 

In [GSj we prove 

(1) ^{TT"^)^pr^,{AV{TT"^))n^ 

(2) *(7r) C *(7r^'^) n z\E'(R), and under certain condition on G cquahty holds. 

1.6. The proposed notion of rank. Wc suggest the following definition of rank. 

Definition 1.6.1. If -k is a smooth Frechet representation of Gn, we define the rank ofn, written rk(7r), 
to be the maximum rank of a matrix in V{tt). 

If V(7r) is a closure of a single orbit given by a partition of length k then rk(7r) = n — k. Theorem [B] 
implies that for tt G G„ our notion of rank agrees with the notion proposed in |Sah89j . and extends Howe's 
notion of rank. In subsection 14.21 we compute (extended) ranks of all irreducible unitary representations 
of Gn in terms of the Vogan classification. 

Our definition extends literally to all classical groups, and connects to Howe's notion of rank by [HeOSi 
Theorem 0.4]. 

For Gn wc can give another interpretation of rank, in terms of parabolic induction. Let A be a partition 
of n, a :~ X- be the inverse reordering of A, Pa the corresponding standard parabolic subgroup. La its 
Levi component. Na its nilradical, and iXq its Lie algebra. Consider the "naive" (non-exact) Jacquet 
restriction functor that maps tt to Tr/riaTr. This functor is adjoint to parabolic induction from Pa- 
Note that W^/i* (tt) is equal to the space Wh* (raiTr)) of generic (classical) Whittakcr functionals on raiir). 
Therefore, Theorem El implies that for it G Gn-, rk(7r) > A: if and only if the Jacquet restriction of tt to 
some Levi subgroup of semi-simple rank k is generic. 

The rk(7r) is also equal to the (real) dimension of the variety consisting of unitary characters ip of N 
such that Wh*X'^) ^ 0. 

For G 7^ Gn we do not have equivalent descriptions of the rank using Jacquet functor or the space of 
functionals. 

1.7. Application to construction of Klyachko models. Our technique can be applied to construction 
of another family of models for unitary representations of G„, called Klyachko models. Let F be a local 
field. For any decomposition n = r -|- 2fc define a subgroup i?r,2fc C G„ , which is a semi-direct product 
of Sp2k{F) and the nilradical N(^i i,2k) of the standard parabolic P(i,...,i.2fe), and let <j)2k.r denote the 
generic character ipi^n) restricted to -/V(i,...,i,2fc) and then extended trivially to -ffr,2fc- Then the r-th 

Klyachko model of tt G Gr+2k is defined to be Hovan^^ATT^ ,4>r,2k)- 

Offcn and Sayag proved existence, uniqueness and disjointness of Klyachko models over p-adic fields. 
They defined the appropriate r in terms of Tadic classification, used derivatives to reduce the statement to 
the case r = 0, in which an S'p-invariant functional is constructed using a global (automorphic) argument. 
Theorem [Bl allows one to define r in terms of the partition corresponding to tt, and then to extend the 
construction of Klyachko models given in jOS07[ §3] to the Archimedean case. This is done in jGOSS] . 
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1.8. Structure of the paper. In section[5]we give several necessary definitions and preliminary results 
on geometry of coadjoint orbits, analytic and smooth vectors, and annihilator varieties. 

In section [3] we prove the main theorem by induction. First we note that 7r|p^ is induced from At:. 
This gives us a map tt°° {Att)°° which has dense image and satisfies a certain equivariant condition 
with respect to a subgroup of P„ that includes N. This also enables us to compute the dimension of 
V(7r|p^) in terms of dimV(A7r). Using the induction hypothesis and the map tt°° {ATr)°° , we show 
that Wh*jj^^^^{TT) ^ 0. By Matumoto's theorem this implies OdcCtt) ^k^Y To show equality we prove 
that the dimension of the annihilator variety does not drop when we restrict tt to P„ , and then use the 
induction hypothesis. 

In section |4] we compute the adduced representation for representations of Speh complementary series 
(except one special case) , thus almost finishing the computation of adduced representations of all unitary 
irreducible representations. We use the restriction on the annihilator variety of the adduced representation 
given by Theorem |^ and a restriction on the infinitesimal character of the adduced representation that 
we deduce from Casselman - Osborne lemma. Those two restrictions allow us to determine the adduced 
representation uniquely, except in one case where we present two possibilities. 

In section [5] we explain how our proofs adapt to the complex case. 

In section IH] we deal with the p-adic case. Most of the arguments here are simply sketched since the 
components of various proofs appear already in jZel80| and |MW87] . 

1.9. Acknowledgements. We wish to thank Avraham Aizcnbud, Joseph Bernstein, Roe Goodman, 
Anthony Joseph, and Nolan Wallach for fruitful discussions and Gordan Savin and the referee for useful 
remarks. 

This paper was conceived when D.G. was a Hill Assistant Professor at Rutgers University. He wishes 
to thank the Mathematics department for warm and fruitful atmosphere and good working conditions. 

2. Preliminaries 

2.1. Induction and dimensions of nilpotent orbits. We now recall Lusztig-Spaltenstein induction 
of nilpotent orbits. Let g be a complex reductive Lie algebra. Let p be a parabolic subalgebra with 
nilradical n and Levi quotient [ := p/n. Then we have a natural projection pr: g* -» p* and a natural 
embedding [* C p*. 

Theorem 2.1.1 f |CoMG93] . Theorem 7.1.1). In the above situation let Oi C I* be a nilpotent orbit. 
Then there exists a unique nilpotent orbit Og that meets pr^^[0[) in an open dense subset. We have 

dim Og = dim Oi + 2 dim n. 

Definition 2.1.2. The orbit Og is denoted Ind^{Oi) and called the induced orbit of 0{. 

Theorem 2.1.3 ( |BB89| . Theorem 2). Let G be a real reductive Lie group. If tt is an irreducible 
representation of G that is parabolically induced from a representation a of a Levi subgroup L, then 
©(tt) = /ndf [O [a]]. 

We now specialize this discussion to g = g„. In this case nilpotent obits are described by partitions. 

Definition 2.1.4. //A,/z are partitions ofp,q of lengths k,l respectively, we define a partition A + /i of 
p + q of length m = max (fc, I) as follows: 

(A + fi), = X, + fi, for i < m 

where the missing Xi,fii are treated as 0. 

Remark 2.1.5. We can describe X+^i using transposed partitions: (A + ^if is the partition rearrangement 
of the composition (A*,/^*). On the level of Young diagrams: X+pi is obtained by concatenating the columns 
of X and pi and reordering them in descending order. 

Proposition 2.1.6 ( |CoMG93] . Lemma 7.2.5). Let X be a partition of I, and pi be a partition of m. Then 

Indl[V;JOxxO^) = Ox+^.. 
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Proposition 2.1.7 ( [CoMG93] . Corollary 7.2.4.). If X is a partition of n and v ^ }\ then 

dimOx ^r? - ^ v]- 

We recall the BZ-product notation for parabolic induction as in Definition 11.1.61 and the associated 
partition AP(t:') as in Definition ll.2.21 From Proposition 12 . 1 .61 we obtain 

Corollary 2.1.8. If a x t is irreducible then 

AP (cr X t) = AP [a) + AP (r) . 

2.2. Unitary induction. Let G be a Lie group and let dx denote a right invariant Haar measure then 
we have 



(g) f (gx) '^^ = J f i^) dx 

where Sq (g) = |det Ad{g)\ is the modular function of G. 

Let H he a closed subgroup of G and write 5h\q (h) = Sh (h) Sg (h) . We define Gc {H\G, Sh\g) to 
be the space of continuous functions f ^ C (G) that satisfy 

(7) / (hx) = dH\G (h) f (x) for all h e H 

and for which supp(/) C HS for some compact set S. Then G acts on Gc (i? \G, 5h\g) by right transla- 
tions and there is a unique continuous G-invariant functional /^^g on Gc (-ff \G, Sh^g) satisfying 



H\G 



SH\G{h) ^f{xh)dh 



H 

-1 



/ (x) dx for all / G Gc (G) 



G 



One shows that f Sh\g f (x^) dh is a continuous surjcction from Gc (G) to Gc {II\G) whose 

kernel is densely spanned by functions / (.t) — Sg (h) f (hx). Consequently dx vanishes on the kernel 
and descends to a functional on Gc [H\G, Sh\g) ■ 

If (a, V) is a unitary representation of H, we define Gc ^iJ\G, S^J^uO^ to be the set of continuous 
functions f : G V that satisfy 

(8) f{hg)^S'J^^{h)a{h)fig) 

and for which supp(/) C HS for some compact set S. For such / we have G Gc {H\G,6h\g) 

and we define W to be the closure of Gc (^H\G, S^^^qO-^ with respect to the norm J^\^q \\f ix)\\Y- Then 
is a Hilbert space under (/i, = Jh\g (/i (f) ' ^^'^ the action of G by right translations 

defines a unitary representation (tt, W) of G. 

Definition 2.2.1. In the above situation (tt, W) is called the unitarily induced representation and denoted 
by Ind%{(j). 

2.3. Analytic and smooth vectors. 

Definition 2.3.1. Let M be an analytic manifold and B be a Banach space. A map M B is said to 
be smooth if it is infinitely differentiate, and analytic if its Taylor series at every point has a positive 
radius of convergence. 

Remark 2.3.2. A map f'.M — > B is smooth/ analytic if and only if the composition (f> o f is 
smooth/ analytic for every continuous linear functional (j): B ^ C . 

Definition 2.3.3. Let (cr, B) be a continuous Banach representation of a Lie group G. A vector v Cz B 
is called smooth/analytic if the action map G ^ B defined by g ^ <^ig)v is smooth/ analytic. Both G 
and its Lie algebra g act on the spaces of smooth and analytic vectors and we denote the corresponding 
representations by {a°°,B°°) and {a'^,B^) respectively. {a'^,B°°) is naturally a Frechet representation 
ofG. 

Remark 2.3.4. By [Nel59[ Theorem 4], B'^ is dense in B. 
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The following theorem by Poulsen can be interpreted as a representation-theoretic version of Sobolev's 
embedding theorem. 

Theorem 2.3.5 f |Pou72| . Theorem 5.1 and Corollary 5.2). Let [tt^W) ^ Ind%{a) ; if f <E W°° (W"^) 
then f is a smooth (analytic) function from G to V . 

Corollary 2.3.6. Let {tt,W) = Ind%{a) he as above; if f € W°° (W^) then f takes values in V°° fV"j. 

Proof. Let / G W°° (VF") and let v := f{g), then the action map of H on u is 

h^a{h)v = a {h) f {g) = 6^\/^ {h) f {kg) 

This is a smooth (analytic) map by Poulscn's theorem; hence v is in {V'^). □ 

Corollary 2.3.7. Let (tt, VK) = Ind%{a); then f ^ /(e) defi. nes a continuous H-equivariant morphism 
{7T°°,W°°) ia°°,V°°) , where a (h) = S]^^^ (h) a {h) as in 0) above. 

The continuity of the evaluation morphism follows from jPou72[ Lemma 5.2]. 

Definition 2.3.8. Let Vi,V2 be two modules for a Lie algebra q; we say Vi,V2 are non- degenerately 
Q-paired if there is a non- degenerate bilinear pairing (•, •) that is invariant in the Lie algebra sense, i.e. 

{Xvi,V2) = - {vi,Xv2} for all vi G Vi,V2 eV2,X £ g 

Lemma 2.3.9. Let {tt,W) — Lnd'fj{a); suppose G/H is connected and a''^ is non-degenerately i)-paired 
with an ^-module r. Then tt" is non-degenerately Q-paired with a quotient ofU{Q) t. 

Here and elsewhere g, f) denote the Lie algebras of G, H and f/(g), U{\)) their enveloping algebras. 

Proof. As noted above g and hence U{q) acts on . Let u' he the principal anti-automorphism of 
U{q) extending X M> —X on g, and define a pairing between tt" and U{q)®t by {f,u®v) := ((w'/)(e), v). 
The pairing descends to U{g) ®)u(t)) ''' in the second variable since if X is in \} then 

(/, uX®v) = - (K {X) u'f] (e) ,v) = - (5- {X) [u' f (e)] , v) = {u' f (e) , r {X) v) ^{f,u® Xv) 

Let us check that the pairing is non-degenerate in the first variable. If / lies in the left kernel, then 
{{u' f)(e),v) = for any u, v and hence / vanishes at e together with all its derivatives. By Theorem 
12.3.51 / is an analytic function and therefore / vanishes in the connected component of e. Since the 
support of / is iJ- invariant and G/H is connected this implies f ~ 0. 

Now if the pairing is degenerate in the second variable, we quotient U{q) (E>u{t)) t by the right kernel. □ 

2.4. Gelfand-Kirillov dimension. Let A be a finitely generated associative algebra over C with in- 
creasing filtration {F''A: i > O}, and let M be a finitely generated A-module. Choose a finite dimensional 
generating subspace S, define a filtration of M by F^M S. It is known that there exists a poly- 

nomial p such that p{i) := dimF*M for large enough i, and that the degree of p does not depend on 
the choice of the finite dimensional generating subspace S. This degree is called the Gelfand-Kirillov 
dimension of M and denoted by GKdim(A'/). 

We will apply this in particular to the case when A is the universal enveloping algebra U{q) for some 
complex Lie algebra g, equipped with the usual filtration inherited from the tensor algebra T (g). 

Lemma 2.4.1 ( | Vog78| , Lemma 2.3). Let t) C g be Lie algebras, and let M be a finitely generated ?7(f)) 
module. Then N = U{g) (8>c/([)) M is a finitely generated U{g) -module, and we have 

GKdim(Ar) = GKdim(M) + dim(g/[)) 

If (cr, M) is a finitely generated g-module, we will sometimes write GKdim(cr) for GKdim(M ) etc. 
Recall the annihilator variety V{a) C g* as defined in 8^5 ^.2.11 It is easy to show that 

GKdim(cr) < dimV(cr). 

There is also a bound in the other direction for the Lie algebras we consider in this paper. 
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Theorem 2.4.2 (Gabber- Joseph). Let g be the Lie algebra of an algebraic group over an algebraically 
closed field. Let M be a finitely generated U{g) module. Then 

GKdim(M) > idimV(M). 

For proof see [Jos81[ Proposition 6.1.4] or |KrLe99[ Theorem 9.11]. This theorem does not hold for 
general Lie algebras. On the other hand a stronger result is true for reductive Lie algebras. 

Theorem 2.4.3 ( |Vog78| , Theorem 1.1). Let G be a real reductive group, g be its complexified Lie algebra 
and K be a maximal compact subgroup. Let [a,M) be an irreducible Harish- Chandra module over {q,K). 
Then 

GKdim(cr) = idimV(cr). 



3. Proof of Theorem [K\ 

We will prove the theorem by induction, using the following three lemmas that will be proven in the 
next three subsections. 

Lemma 3.0.1. Let tt e Gn and let d := depth{'K). Let a = (ni, Uk) be a composition of n — d and 
13 = (d, a) = (d, 711,71,2, ...,7ifc). Then we have a natural embedding VF/i*(A7r) ^ VF/i^(7r). 

Lemma 3.0.2. Let X, fi be partitions of n and n~ d respectively, and suppose that 

(1) c Wx. 

(2) dim Ox < dim 0^ + (2n - d) (d - 1) . 
Then X = (d, fi) . 

Lemma 3.0.3. Let tt e G„ and let d depth{TT). Then 

dimV(7r) < dim V(A7r) + (277 - d)(d - 1). 

Proof of Theorem[A\ We prove the statement by induction on n. For 7i = 0, 1 there is nothing to 
prove. Now take 7i > 1 and suppose that the theorem holds true for all r < n. Let tt G G„, d := 
depth{Ti) and A := AP{tt). 

Let ^ be the depth composition of At:. By the induction hypothesis we know that /i is a partition of 
n - d, V{Att) = and WhliAir^) ^ 0. 

Let /3 — {d,fi). Then /3 is the depth composition of tt. From the induction hypothesis and Lemma 
IHini wc obtain iyft,^(7r°°) ^ 0. It suffices to show that /? A. 

By Corollary [HO we have O/3 C V(7r) = O^. By Lemma we have dimOA < dimO^j + 

(277 - d) (d - 1) . Thus, by Lemma [2121 /3 = A. □ 

3.1. Proof of Lemma l3.0.1l Now we construct a functional on tt G G„ from a functional on its adduced 
representation. 

Let P(n-d,d) = (Gn-d X Gd) K N(ji-d,d) bc the maximal parabolic subgroup corresponding to the 
partition (77 — d, d) and define the subgroup Sn-d,d '■= {Gn-d x ^i<i) ^N(^n-d.d) where Nid is the nilradical 
of the Borel subgroup of Gd. 

Lemma 3.1.1. Let tt G G„, and let d :~ depth{n). Then 7r|p^ — Lndg" ^ ^ {An (g) t/^ (3 1) where ^ is a 
nondegenerate unitary character of Nid . 

Proof. This follows from the definition of At: by a straightforward argument involving induction by 
stages. □ 

Proposition 3.1.2. Let tt G G„, and let d :~ depth{Tr). Then there exists an Sn-d.d-^quivariant map 
from TT°° to {Att)°^ (g) | dot [('^^i)/^ with dense image. 
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Proof. Note that for g e Sn-dM, Ap„(5) = |det(5)| and As,^_^ ^{g) ^ |det(5)|''. By Corollary [1X71 the 
evaluation f ^ f {e) is an 5„_d,d-equivariant map from (7r|p^)°° to {Att)°° (g) | det j^''^^)/^. Note that 
An is an irreducible unitary representation of Gd and thus, by jWall88[ Theorem 3.4.12], {ATr)°° is an 
irreducible Frechet representation of Gd- Thus it is enough to show that this map does not vanish on 7r°°. 
For that let v € 7r°° be a non-zero vector. Then v G 7r|^ = Indg" ^ ^ {Att ®'ip 'Si 1)°° defines a smooth 
function on that does not vanish at some point p. Then tt{p)v does not vanish at e. □ 

This proposition immediately implies Lemma 13.0.11 

3.2. Proof of Lemma 13.0.21 It will be useful to have a second formula for dimOA directly in terms of 
A. We will also consider rcordcrings of A. 

Lemma 3.2.1. If a is a composition of n then we have 

(9) dim Oa > + n - 2 ^ iai. 

Moreover, equality holds if and only if a is a partition. 

Proof. Let A = a- and let i/ ~ he the transposed partition. 

We first show that equality holds in © if o; is a partition, i.e. a ~ X. By Proposition 1 2 . 1 . 7l it suffices to 
prove 

E,^-25:^.A,-n. 

Consider the Young diagram of A and write the number 2i in every box in the i-th row. Compute the 
sum of these numbers in two ways: a) adding rows first, b) adding columns first. This gives 

as needed. 

Now, for general a we have Oa = Ox. If a is not a partition then we have strict inequality in ([9]) since 

Corollary 3.2.2. Let fi be a partition of n — d and write a = (d, m) then we have 

dim Oa > dim + (2??, - d) (d - 1) , 
and equality holds if and only if a = (rf, yu) is a partition (i.e. d > ^li). 
Proof. We observe that 

E = d+ E. + 1) = " + E. 

Let r ^ n — d and apply Lemma 13.2.11 to get the following inequality 

dim Oa - dim > (^n^ + n - 2 ^ ja^J - (^r^ + r - 2 ^ ifi, 

= (n^ - 77.) - (7-^ + r) = (77 + r) (77 - 7- - 1) = (277 - d) {d - 1) 
By Lemma 13.2.11 cqualitv holds iff a is a partition. □ 

Proof of Lemma \3.U.2\ Let us write a = {d,fi). By assumptions (1) and (2) and CoroUarv 13 . 2 . 21 we get 

dim Oa < dim Ox < dim O,, + (2r7 - d) {d - 1) < dim 

Hence equality must hold throughout. This implies that dimOa = dimOA and, by CoroUarv 13.2.21 
that a is a partition. Now assumption (1) implies that a = X. □ 
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3.3. Proof of Lemma 13.0.31 First we want to prove that dim V(7r|p^) > dimV(7r). We will start with 
a geometric lemma. 

Lemma 3.3.1. Let O C g* be a nilpotent coadjoint orbit. Then there exists an open dense subset U <Z O 
such that the restriction to U of the projection pr pr^* is injective. 

For the proof see iiS il3.3.4l 

Corollary 3.3.2. Let V C g* be the closure of a nilpotent coadjoint orbit. Then dimprpr(V) = dimV. 

Corollary 3.3.3. Let tt E G^. Then dim V(7r|p„) > dim V(7r). 

Proof. Let / = ylnn(7(g^)7r" and J — 74nn(7(p^) (7r|p^ Since (7r|p,J" D tt" we have J C An?i(7(p^)7r" = 
/ n U(pn) and hence we get 

gr{J) C gr{I n C/(p„)) C gr{I) n 5(p„). 
Since V{tt) is the closure of a nilpotent coadjoint orbit, by the previous Corollary we conclude 

dimV(7r|p„) > dim Zeroes (gr(/) n S{pn)) = dimprp? (V(7r)) = dimV(7r). 

□ 

Now wc want to use Theorem l2.4.2l to bound dim V(7r|p^). In order to do that we find a finitely gener- 
ated U (p„)-module which is non-degenerately paired with (7r|p^J" and therefore has the same annihilator. 

Proof of Lemma\MIE We wiU use Lemma [5XT] Let a := Att (g) ■0 ® 1 and t | det |(i-'i)/2((^7r)ffc ^ 
■0(8)1) be representations of Sn-k,k and Sn-k,k respectively, where (An)^'-^ denotes the complex conjugate 
representation to (Att)^''^ . Then r is equivariantly non-degenerately paired with a = | det |'^''~^^/^cr. 

By Lemma |3.1.1[ tt\p^ = Ind^" ^i'^) ^''^d thus, by Lemma 12.3.91 (7r|p^)" is equivariantly non- 
degenerately paired with a quotient of U{pn) ®u{5„~k k) Denote this quotient by L. Then V(7r|p^) = 
V(L). Note that twist by a character does not effect annihilator variety and Gelfand-Kirillov dimension. 
Neither does exterior tensor product with a character. 

From Lemma 12.4.11 we obtain : 

GKdim(L) < GKdim(T) + dimp„ - dims„_fe,fe = GKdim(A7r^<^) + (2n - d){d - l)/2. 

By Theorem HXl we have dimV(L) < 2GKdim(L). By Theorem [2X3] we have 2 GKdim(A7r^'^) = 
dimV(A7r). By Corollarv 13.3.31 we have dim V(7r) < dimV(7r|p^). Altogether we have 

dimV(7r) < dimV(7r|p„) = dimV(i) < 2 GKdim(L) < 

< 2GKdim(A7r^'^) + {2n - d){d - 1) = dimV(A7r) + {2n - d){d - 1). 

□ 

3.3.4. Proof of Lemma \3. 3.1\ Identify g* with g„ using the trace form; then p* consists of matrices whose 
last column is zero, and pr {A) replaces the last column of A by zero. Now O corresponds to a nilpotent 
orbit in g„ ; let k denote the size of the biggest Jordan block in O and define 

U ■.^{AeO\ e'^A^-^ ^ 0} where = (0, • ■ • , 0, 1) 

Then U is an open dense subset of 0, and we will show pr\ij is injective. Suppose A, B E U with 
pr (A) — pr {B), then A and B differ only in the last column and so 

B ^ A + ve'^ 

where v is some column vector; it suffices to prove that w = 0. 

We first prove by induction that A*w = for any i > 0. Since A and B are nilpotent we have 

= Tr i? = Tr ^ + Tr ue'„ = e^ji; 

which proves the claim for i = 0. Now suppose the claim holds for i < I, then 

(10) B'+^ = {A + ve'J+^ =A'+^ A^ve'J A'-^ + ■ ■ ■ 
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Each omitted term in ([TU]) has at least two factors of the form ve'^, hence at least one factor of the form 
e'j^A^v for some < i < I , which vanishes by the induction hypothesis. Now taking trace in ([TU| we get 

- E -.0 = + 1) ^n^'^ 

which implies the claim for i = I, and by induction for all i. 

Suppose now by way of contradiction that v ^ and let m > be the largest integer such that 
A^v ^ 0. Substitute ? = fc - 1 in since = B'' = we get 

(11) = V'^'"^ A^ve'A''-^-^ = V" A^ ve'A''-^-^ 

Note that v,Av,...,A™v are linearly independent; indeed suppose X^JLiCjA-'?; = with Ci ^ 0, then 
multiplying by A™^* we deduce CiA™u = 0, which is a contradiction. Therefore we can choose a row 
vector (f) such that (pA^v = 1 but (pA^v = for any j < m. Multiplying ([TT|) by </> on the left we get 
= e'^A^~^~"^ which contradicts the assumption that eJ^A'"'^^ 7^ 0. □ 

Remark 3.3.5. Using |Ber84[ 3.1;4.1-4.2] one can show that U is a single Pn-orbit. 

4. Computation of adduced representations for (almost) all unitary representations 

It is an interesting and important problem to explicitly compute the adduced representations of all 
irreducible unitary representations of Gn- The answer has been conjectured in |SaSt90| and in the present 
paper we make substantial progress towards the proof of this conjecture. 

4.1. Vogan Classification. By the Vogan classification |Vog86| , irreducible unitary representations of 
Gn are Bernstcin-Zelevinsky (BZ) products of the form 

TT = TTl X ■ • • X TTfc 

where each tt^ is one of the following basic unitary representations: 

(a) A one- dimensional unitary character of some Gm- Such a character is of the form 

X M> (sgndct xY |det x|" , £ £ {0, 1} , z e C 
and we shall denote it by x ('^j z)- This character is unitary if z is imaginary, i.e. 

z = e K. 

(b) A Stein complementary series representation of some G2nn twisted by a unitary character. The Stein 
representations are complementary series of the form 

cr (2m, s) = x (m, 0, s) X X {m, 0, -s) , s £ (0, 1/2) 

and we write a (2m, s; e, it) to denote its twist byx (2m, e, it). 

(c) A Speh representation of some G2m, twisted by a unitary character. As shown in |BSS90| and 
jSaStQOj the Speh representation 6 (2m, k) is the unique irreducible submodule of 

(12) X {m, 0, fc/2) X X (™, £fc+i, -k/2) , fc e N, Sk+i = k + l (mod 2) 

and we write 5 (2m, fc; it) to denote its twist by x (2m, 0, it). 

(d) A Speh complementary series representation of some G^m, twisted by a unitary character. The Speh 
complementary series representation is 

(4m, fc, s) = (5 (2m, fc; 0,s)xS (2m, fc; 0, -s) , s G (0, 1/2) 

and we write ijj (4m, k, s; it) to denote its twist by x (4w, 0, it). 

The reader might well ask why we do not consider twists in (jcj) and ([d]) for e = 1. The reason is the 
following: 

Lemma 4.1.1. Speh representation and their complementary series are unchanged if we twist them by 
the sign character. 
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Proof. If TT is a representation of G„ we denote its sign twist by tt ~ tt x (f^-, 1, 0). This operation is 
compatible with parabolic induction in the sense that we have 

(13) TTl X 772 = TtT X TtJ. 

We leave the easy verification of (jl3|) to the reader. 

For the group GL (2,M), the Speh representations S (2, k) are precisely the discrete series. In this case 

the result 5 (2, k) = 5 (2, k) is well known (see 1.4.7 in |Vog81[ ). The general Speh representation S (2m, k) 
is the unique irreducible quotient (Langlands quotient) of 

(14) (5 (2, fc; si) X • • • X (5 (2, k; s,„) where Si = ~ * 

see, e.g. |Vog86| . By (fT3|) the induced representation (fT4|) is unchanged under sign twist. Therefore so is 
its unique irreducible quotient. 

The result for Speh complementary series now follows from (|13p . □ 

4.2. Annihilator variety and rank. In this subsection we compute the associated partition in terms 
of the Vogan classification. Let 

(15) TT = ^ X, X []^.^^ S, with n = ^'^^ + 2 Y,]^^ 1j 

where Xi is a character of Gp- and 6j is a Speh representation of G2q (perhaps twisted by a nonunitary 
character in order to include complementary series). By Corollary 12.1.81 . to compute AP{Tr) it suffices 
to determine AP{x^) and AP{dj). Clearly AP{xt) = V- By |SaSt90[ Theorem 3], the adduced of the 
Speh representation S{2n,k) G G2n is the Speh representation S{2{n — l),fc) G G2n-2- By induction, we 
obtain from Theorem El that AP{6j) ^ 

Theorem 4.2.1. For tt as in \15\) . AP{'kY has one part of size pi for each i and two parts of size q.j for 
each j. Consequently the rank (see Definition \1.6.l\) and the Gelfand-Kirillov dimension of n are given 
by 

(16) rk(7r) = n - max(p,;, g,), GKdim(7r) = ^ - " ^ ^ 1j) 

Corollary 4.2.2. Let tt G Gn and let k < n/2. Then rk(7r) — k if and only if there exist a representation 
cr G Gk and a character % G Gn-k such that n = a x x- 

Remark 4.2.3. By |Sca90[ Part II, Corollary 3.2], an analogous statement holds for Howe rank. This 
gives an independent proof of the statement that for any n G G„, 

Howe rank (tt) = min(rk(7r), \ n/2\). 

The following theorem follows from |BSS90[ [SaSt90] . 

Theorem 4.2.4. For the Speh representation J(2n, k) G G2n there exist degenerate principal series 
representations iTk and 7r_fc, induced from certain characters of the standard parabolic subgroup given by 
the partition (n, n), an embedding i: S(2n, k) tt^. and an epimorphism p: 7r_fe -» d{2n, k). 

Corollary 4.2.5. Any irreducible unitarizable representation n of Gn can be presented both as a subrep- 
resentation and as a quotient of a degenerate principal series representation with the same annihilator 
variety. Those degenerate principal series representations will be induced from characters of the standard 
parabolic described by the partition which is transposed to the partition describing V(7r). 

Remark 4.2.6. In |AGS| we show, using |BSS90| . that all other Jordan-Holder constituents of the 
degenerate principal series representations mentioned above will have smaller annihilator varieties. 
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4.3. Infinitesimal characters - general considerations. In this subsection only, we let q denote 
an arbitrary complex reductive Lie algebra. We fix a Cartan subalgebra f) and a Borel subalgebra 
b = [) + n, and let W denote the Weyl group of g. Let Z (g) denote the center of the universal enveloping 
algebra U (g). Then by the Harish- Chandra homomorphism we have Z (q) fa S (i))^ ■ Thus each A G f)* 
determines a character x\ of Z (g) with Xw\ = X\ for all w G W. 

The Harish- Chandra homomorphism is a special case of the following more general construction. Let 
q = I + u be a standard parabolic subalgebra containing b so that f) C I and u C n. Then we have a 
triangular decomposition g = u + [ + u, and by the PBW theorem the universal enveloping algebra of g 
can be decomposed as follows: 

(17) U {q) = U {u) ®U {{) ®U {u) = U ([) © [uU (g) +U (g) u] 

Let V = V^ denote the corresponding projection from U (g) to U (I), and let Z (g) and Z ([) denote the 
centers of U (g) and U (I). 

Lemma 4.3.1. (1) V is ad {l)-equivariant. 

(2) V maps Z {q) to Z{{). 

(3) For z € Z{q) we have z - P (z) e uU (g). 

Proof. See e.g. |Vog81| P. 118. □ 

Lemma 4.3.2. Let V be a g-module, then 

(1) The subspace uV is Z (q) -invariant and U (V) -invariant. 

(2) The quotient space V/uV is a Z {Q)-module and a U (I) -module. 

(3) For z Z (q) the actions of z and P (z) agree on V/uV . 

Proof. Parts 1 and 2 are straightforward. Part 3 follows from the previous lemma. □ 

We say that a g-module V has infinitesimal character A if each z ^ Z {g) acts by the scalar x\ {z). We 
say that V has generalized infinitesimal character x\ if there is an integer n such that (z — x\ (z))" acts 
by for all z G Z (g). We say that V is Z (g)-finite if V is annihilated by an ideal of finite codimension 
in Z (g). li V is Z (g)-finitc then V decomposes as a finite direct sum 

V = Vi®---®Vk 
where each Vi has generalized infinitesimal character. 

Corollary 4.3.3 (Cassclman - Osborne). Let V be a Z (g)- finite g-module, then V/uV is a Z (V)- finite 
{-module. Moreover if the generalized infinitesimal character Xp. occurs in V/uV , then there exists A G ()* 
such that 

(1) the generalized infinitesimal character x\ occurs in V ; 

(2) /X = A + p (u) where p (u) is half the sum of the roots of \) in u. 

Proof. The proof is similar to that of Corollary 3.1.6 in |Vog81| . □ 

4.4. Infinitesimal characters for G„. If tt is an irreducible admissible representation of G„, the in- 
finitesimal character of can be regarded as a multiset (set with multiplicity) of n complex numbers 
{zi, . . . , z„}. If all the Zi arc real we say that tt has real infinitesimal character. For convenience we write 
U for the (disjoint) union of multisets; e.g. {1, 2} U {2, 3} = {1, 2, 2, 3} . 

For TO G N and z G C we define the corresponding "segment" to be the set 

S (to, z) = {zi, • • • , z,„} where zi ~ z + (to. + 1) /2 — k. 

Thus zi, ■ • ■ , Zm is an arithmetic progression of length to, mean z, and common difference 1. 

The following lemma summarizes the main facts about infinitesimal characters for unitary representa- 
tions of G„. 

Lemma 4.4.1. (1) ^yrix---x7r^ = U • • • U ^^^^ 

(2) For TT = X ("^; z) we have ^tt ~ S (to, z) 

(3) For TT = a (2to, s; e, it) we have 5,,^ ~ S (to, s + it) Li S (to, — s + it) 
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(4 ) For TT = 5 (2m, fc; it) we have = S (m, (fc/2) + it)\JS (m, (-fc/2) + it) 

( 5) For TT = ?/' (4™, k, s; it) we have ^tt = \_\S (m, ± (fc/2) ± s + it) 

Proof. 1) and 2) are standard |Vog81| and together they imply 3). Similarly 1) and 4) imply 5). Part 
4) follows from 1) and formula (|12p : alternatively one may deduce it fr-om 1), formula (HH), and the fact 
that the discrete series S (2, fc) of GL (2,R) has infinitesimal character {f, — f}- D 

Lemma 4.4.2. If tt G G„ then is symmetric in the sense z and —z have the same multiplicity in 
for all z G C. 

This lemma is in fact an easy elementary fact about Hermitian representations. For tt G G„ it also 
follows easily from the previous lemma by checking it for basic representations. 

4.5. Adduced representations. 

Conjecture 1 f [SaSt90] ). Let tt G Gn and write tt = tti x • • ■ x tt^ as in the Vogan classification with 
each TTi a basic unitary representation of type listed above. Then we have 

(18) Att = Am X • • • X Awk 

where 

(19) A{xim,e,it)) ^ x{m~ l,e,it) 

A {a (2m, s; e, it)) = cr (2 (in — 1) , s; e, it) 

A {5 {2m, fc; it)) = (5 (2 (to - 1) , fc; it) 
A {ip (4to, fc, s; it)) = -0 (4 (m - 1) , fc, s; it) 

Most of this result is already known. The identity ^TE\\ is proved in |Sah89| . As for (fTO)) . part 1 is 
obvious, part 2 is proved in |Sah90j . part 3 is proved in |SaSt90| . where part 4 was conjectured. We show 
that the techniques of the present paper suffice to prove ((T9)) part 4 for fc 7^ to. 

Lemma 4.5.1. Let tt G Gn and for i G M tei Tr[it] denote the unitary twist of tt by the character jdetf*. 
Then we have A{TT[it]) ~ (Att) [it]. 

Proof. This is straightforward. □ 

Lemma 4.5.2. The Speh complementary series representation ijj {Am, k, s]it) is uniquely determined by 
its infinitesimal character and associated partition. 

Proof. By Lemma [4.4.11 tt = if) {Am, k, s; it) has infinitesimal character ~ |J S" (to, ± (fc/2) ± s + it), 
and by Theorem 14.2. II its associated partition is 4™. Since 0<s<l/2it follows that 

(1) 2 Re (z) is not an integer for any z G ^tt, 

(2) max{2Re(z) : z G Ctt} = - 1 + fc + 2s 

Let tt' be a unitary representation with the same infinitesimal character and associated partition as 
TT. Write tt' = TTi X • • • X TT; as in the Vogan classification. Then condition 1 above implies that none of 
the TTi can be unitary characters or Speh representations, while condition 2 implies that tt' cannot be the 
product of two Stein representations of G^m, for then we would have max (2 Re (z)) < to. Therefore we 
conclude that tt' is a Speh complementary series representation of Gim- Thus tt' = -0 (4to, fc', s'; it'). By 
looking at the integral and fractional parts of max (2 Re (z)) we deduce k = k' and s = s' . By looking at 
the imaginary part of z G = S.-n' we conclude that t = t' . □ 

Remark 4.5.3. Due to the previous lemma, Conjecture]^ becomes now equivalent to the statement that 
for any tt G Gn, the infinitesimal character parameter of Att is obtained from the infinitesimal character 
parameter of tt by the following procedure: consider the Young diagram X with the sizes of rows described 
by the partition corresponding to tt, write the infinitesimal character parameter of tt in the columns such 
that in each column we will have a segment, and make each of those segments shorter by one without 
changing its center. This is similar to the effect of highest derivatives on the Zelevinsky classification in 
the p-adic case (see ^dj). We cannot prove this statement here, but we deduce its weaker version from 
Corollary \4.S.3\ and Proposition IS.l.'d 
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Proposition 4.5.4. Let tt G Gn and let d = depth{Tr). Let S = {zi, . . . , Zn} and 5' ~ {j/i, . . . , yn-d} be 
the multisets corresponding to infinitesimal characters of t: and At: respectively. Then S'is obtained from 
S by deleting d of the Zi 's and adding 1/2 to each of the remaining Zi 's. 

Proof. Let cr := Att and let 7r°° and ct°° be the spaces of smooth vectors of tt and cr, respectively, and let 
A and /i be their infinitesimal character parameters. 

By Proposition 13. 1 . 2l there is an 5,1 _£i,d-equi variant morphism ip : 7r°° — > a°° ® \ det [(''^i)/^ with dense 
image. Since Nn^d.d acts trivially on (t°° ® | det [(''"i)/^^ (p factors through the quotient 7r°°/u7r°°, where 
u = Lie {Nn-d,d)- By Corollarv 14.3.31 the possible Gn-d x Gd infinitesimal characters in 7r°°/u7r°° are of 
the form wX + p{u). Further restricting to Gn-d we conclude that these characters are of the form 

A- + ^(l,l,---,l) 

where C A is a subset of size n — d. 

On the other hand, the Gn~d module (t°° ® \ det has infinitesimal character 

d-l , 

Comparing the two displayed expressions we get ^ = A^ + i(l,l, •••,!) and the result follows. □ 



We are now ready to prove the main theorem of this section. 
Theorem 4.5.5. Suppose k ^ m then A (-0 (4m, k, s; it)) — ip {4 {m ~ 1) , k, s; it). 

Proof. Let it = ip (4m, fc, s; it) , tt' = (4 (m — 1) , A:, s; it) and let ^, ^ be the infinitesimal characters of 
7r,7r', Att respectively. By TheoremSXH AP{tt) = 4™ and APi-n') = 4™-i. By TheoremEl AP{Ati) = 
4™"-'^ as well. Therefore by Lemma [4.5.21 it suffices to show that — ^ . 

For simplicity we assume < = 0, the argument is the same if t 7^ 0. Now by Lemma 14.4.11 we have 

i = [}s(^m,±^^±s^ ande' = |j5(^m-l,±^±s^ 

Now bv l4.5.4l f is obtained from _B = ^ + ^ by deleting 4 elements. Indeed ^' is obtained from B by 
deleting the following 4 elements 

^2 2 ^ 

and we need to show that no other infinitesimal character of a unitary representation can be so obtained. 

In fact we note that Sf is the only symmetric sub multiset of ^ + ^ with |^'| = 4 (m — 1). Indeed any 
other symmetric subset of cardinality \B\ — 4 is obtained from B by replacing a symmetric subset of Sf 
with a symmetric subset of equal size contained in C, but \i k ^ m then G has no symmetric subsets. □ 



Remark 4.5.6. For k = m the proof of Theorem \4.5.5\ fails, since in this case we have 

G — {m ± s, ±s} 

which admits the symmetric subset {±s} . Lndeed it is easy to see that for k = m the infinitesimal character 
of tp (4m — 4, m — 1, i — s) is also a subset of ^ -\- ^. Therefore we may conclude only that 

Alp (4m, m,s) = ip (4to — 4, m, s) or ip ^4m — 4, m — 1, — — 
An additional argument is required to rule out the latter possibility. 
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5. The complex case 
Let us discuss the setting and the geometry of nilpotent orbits. We have 

G = G„ = GL(?2,C), 0«0[(n,C)®0[(n,C), 0r = gl^ C) © 0, e = {(a;,a;)}, «^ = {(x, -a;)}, 

where 6 denotes the complexified Lie algebra of maximal compact subgroup. The nilpotent orbits are 
parameterized by pairs of partitions and have the form Oa x O^. However, associated orbits of Harish- 
Chandra modules intersect t-^ and are therefore of the form 0\ x Ox and so are still parameterized 
by single partitions, rather than two partitions. Standard parabolic subgroups are also parameterized 
by single partitions. The degenerate Whittaker functionals are defined in the same way as in i)i) ijl.2.3l 
using Ja € 0R. Therefore they are also parameterized by single partitions. Therefore the formulation of 
Theorem 1X1 in the complex case stays the same. The proof of Theorem 1X1 is obtained from the proof in 
Sj3]by replacing the term "coadjoint nilpotent orbit" by "coadjoint nilpotent orbit that intersects 6"*"" and 
doubling all the expressions for the dimensions of such orbits. 

The Vogan classification of GL(n,C) is simpler: each tt G GL{n,C) is a product of characters, where 
the non- unitary characters come in pairs. As shown in jSah89[ I5ah90j . the adduced representation of tt is 
a product of the same form, where each character of Gn is restricted to G„_i, and characters of Gi are 
thrown away. The associated partition of tt is determined in the obvious way, similar to Theorem 14.2.11 

6. The p-ADic case 

In this section we fix to be a non- Archimedean local field of characteristic zero and let G„ := 
GL{n,F). 

6.1. Definition of derivatives. In the p-adic case, there is an additional definition of highest derivative, 
using CO- invariants. This definition works for all smooth admissible representations of G„, not only 
unitarizable. Moreover, Bernstein and Zelevinsky define in jBZ77[ §3] all derivatives and not only the 
highest ones in the following way. 

Recall that P„ = G„_i k Fix a non-trivial additive character x of F and define a character 9n 

of F""^ by applying x to the last coordinate. Denote by i/ the determinant character 1/(17) = | det(f;)|. 

Define two normalized coinvariants functors : Rep{Pn) Rep{Gn~i) and $^ : Rep{Pn) 
Rep{Pn-i) by 

^'-(t) := i/"^/^T^„-i_i and $"(t) := v^^/'^Tpr.-i q^. 

Both functors are exact. 

For a smooth representation r of P„ they define r^*^' ^'^($~)'^~^t and call it the fc-th derivative of 
T. For a smooth representation tt of G„, k-th derivative is defined by D^t: := n^''^ := {'t\p^)'^''\ 

If D'^tt > but D^^^-K = for any / > then D'^tt is called the highest derivative of tt and we denote 
it by A{t:), and k is called the depth of tt and denoted d(Tr). 

For unitarizable representations, one can also define a shifted highest derivative of tt using Mackcy 
theory, in the same way as adduced representation is defined in the Archimedean case (see i) tjl.2.6p . By 
[Ber84j . this shifted highest derivative will be isomorphic to v-^^^AItt). 

For a composition a = (ai,...,afc) we define D^i?:) := D°'K..D°"' {?:). By jZel80[ §§8.3] we have 
(D«'=D°^-i...D«i7r)* ~ Whliir). 

Denote by A4 {Gn ) the category of smooth admissible representations of G„ and define the Grothendieck 
ring R = 0„r(A^(G„)). As an additive group it is the direct sum of Grothendieck groups of A4{Gn) 
for all 71, and the product is defined by parabolic induction. Define the total derivative D : R ^ R as the 
sum of all derivatives. By |BZ77I §§4.5], D is a homomorphism of rings. 

6.2. Zelevinsky classification. In [Zel80| . Zelevinsky describes the generators of R in the following way. 
Denote by G 1J„ G„ the subset of all cuspidal irreducible representations of G„ for all n. For p G G^, 
a subset A C Gd of the form {p,vp,v'^ p, ...,v'-~^ p) is called a segment. The representation is 
called the center of A, the number I is called the length of A, and the number d is called the depth of A. 
We denote the set of all segments A C G by We define a segment A~ by A^ = {p, vp, v^p, v'-^'^p). 
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Theorem 6.2.1 f |Zel80| .^3 and §§7.5). Let A = {p.vp.v'^ p, ...,v^-^ p) C Cd be a segment. Then the 
representation px i/px ...v^~^p contains a unique irreducible constituent (A) of the depth d = depth{A). 
Moreover, 

(1) i?((A)) = (A) + (A'). 

(2) R is a polynomial ring in indeterminates {(A) : A G 5} 

Zelevinsky furthermore describes all irreducible representations in terms of segments and shows, in 
|Zel80[ §§8.1], that A maps irreducible representations to irreducible. 

6.3. The wave front set. Let tt G A4(G„). Let x-k be the character of tt. Then Xtt defines a distribution 
on a neighborhood of zero in g„ . by restriction to a neighborhood of 1 G G and applying logarithm. This 

distribution is known to be a combination of Fourier transforms of Haar measures of nilpotent coadjoint 
orbits ( |HC77i p. 180]). This enables to define a wave front cycle, WFC{tt) as a linear combination 
of orbits. Clearly, WFC is additive on R. In |MW87[ II. 1] it is shown that it is also multiplicative, in 
the sense of CoroUarv 12.1.81 The wave front set, WF{tt), is defined to be the set of orbits that appear 
in WFC{tt) with non-zero coefficients. Denote by WFmax{TT) the set of maximal elements of WF{tt) 
(with respect to the Bruhat ordering). In jMW87[ II. 2], WFC {(A)) is computed to be the orbit given by 
the partition Aa that has length{A) parts of size depth{A), with multiplicity 1. Thus, for an arbitrary 
smooth irreducible representation tt of G„, WFmax(Tr) consists of a single nilpotent orbit, given by the 
partition defined by the Zelevinsky classification (see |OS09j ). 

6.4. Proof of Theorem |B] over F. Since both D and in a sense WFC are homomorphisms, and 
since segment representations (A) generate R, it is enough to prove Theorem |B] for them. The above 
information implies that Wha{{A)) = £'"((A)) unless a Aa- Now, Whx^i{A)) = /^^^(A) = 
(0) = C. Since W^i^G((A)) = 0\^, this completes the proof of the main three parts of Theorem iBl 

We can prove the "moreover" part through an analog of Corollarv l4.2.2l there is a Tadic classification 
of the unitary representations as products of certain building blocks (see ITadSOj ). from which we see that 
rank^n) = k < n/2 if and only if tt = % x r, for some character x of Cn-k and some r G M.{Ck)- Since 
the same holds for Howe rank (by |Sca90[ Part II, Corollary 3.2]), the "moreover" part follows. One can 
also prove the "moreover" part directly, as was done in jMW87l II. 3] for the symplectic group. 
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